A Conversational Model of Multimodal Interaction

Multimodal interaction is employed in a variety of contemporary information systems in order to enhance the flexibility and naturalness of the user interface. In this paper, we propose a comprehensive framework which allows the modeling and specification of multimodal interactions. To this end, we employ an extended notion of ‘dialogue acts’ which can be performed by linguistic or non-linguistic means. On this basis we discuss the temporal structure of multimodal interaction. First, a comprehensive set of constraints is presented that describes all patterns of exchange which can be encountered during a cooperative information-seeking dialogue. Second, we introduce a strategic level of description, which allows the specification of the topical structure of the dialogue according to a selected information-seeking strategy. The model was used to design and implement the MERIT system (Multimedia Extensions to Retrieval Interaction Tools), and led to a reduction in the complexity of the user interface while preserving most of the useful, but sometimes confusing, dialogue options of advanced direct manipulative interfaces. An abbreviated version is published in: Proc. of the 11th National Conference on Artificial Intelligence (AAAI ’93), Washington DC, USA, July 11-16, 1993. Menlo Park: AAAI Press/ The MIT Press, 1993, pp. 283-288. Also available as: Arbeitspapiere der GMD No. 741, Sankt Augustin: GMD, March 1993. Address all correspondence to: Dr. Adelheit Stein, Dr. Ulrich Thiel, GMD-IPSI, Dolivostraße 15, D-64293 Darmstadt E-mail: [email protected] Phone: ++ 49 / 6151 / 869-841 2 A Conversational Model of Multimodal Interaction Stein, Thiel